Papers with self-attention layers
Efficient Content-Based Sparse Attention with Routing Transformers (2021.tacl-1)
Copied to clipboard
| Challenge: | Self-attention suffers from quadratic computation and memory requirements with respect to sequence length . despite its effectiveness, self-attention models suffer from quadratic computation and a limited set of locations . |
| Approach: | They propose to learn dynamic sparse attention patterns that avoid allocating computation and memory to attend to content unrelated to the query of interest. |
| Outcome: | The proposed model outperforms similar sparse attention models on language modeling and image generation on Wikitext-103 . |
A Non-Linear Structural Probe (2021.naacl-main)
Copied to clipboard
| Challenge: | Probing is a method of investigating the encoding of knowledge in contextual representations. |
| Approach: | They propose to kernelize a metric and develop a non-linear variant with an identical number of parameters by using a kernel-based probe. |
| Outcome: | The proposed probe learns only linear transformations and achieves statistically significant performance improvement over baseline in all languages. |
Towards efficient self-supervised representation learning in speech processing (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing models require several GPUs for days to pretrain, generating environmental concerns because of their high energy consumption. |
| Approach: | They propose an efficient self-supervised model that uses a single GPU during 24 to 48 hours of pretraining to address high computational costs. |
| Outcome: | The proposed model represents two orders of magnitude better than existing models. |
Rethinking Self-Attention: Towards Interpretability in Neural Parsing (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work shows that attention mechanisms provide arguably explainable attention distributions that can help to interpret predictions. |
| Approach: | They propose a new self-attention layer where attention heads represent labels. |
| Outcome: | The proposed model obtains state-of-the-art results on the Penn Treebank and Chinese Treebank. |
Lattice-BERT: Leveraging Multi-Granularity Representations in Chinese Pre-trained Language Models (2021.naacl-main)
Copied to clipboard
| Challenge: | Pre-trained language models process text as a sequence of characters, ignoring more coarse granularity, e.g., words. |
| Approach: | They propose a new pre-training paradigm for Chinese that incorporates word representations along with characters and can model a sentence in a multi-granular manner. |
| Outcome: | The proposed model can bring an average increase of 1.5% under the 12-layer setting, which achieves new state-of-the-art among base-size models on the CLUE benchmarks. |
Double Path Networks for Sequence to Sequence Learning (C18-1)
Copied to clipboard
| Challenge: | Existing approaches for Sequence to Sequence learning have been developed . convolutional neural networks and self-attention networks are the most popular . |
| Approach: | They propose to integrate convolutional and self-attention layers into a double path network for sequence to sequence learning. |
| Outcome: | The proposed method significantly improves performance over state-of-the-art systems. |
Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization (2023.acl-long)
Copied to clipboard
Pengcheng He, Baolin Peng, Song Wang, Yang Liu, Ruochen Xu, Hany Hassan, Yu Shi, Chenguang Zhu, Wayne Xiong, Michael Zeng, Jianfeng Gao, Xuedong Huang
| Challenge: | Z-Code++ is a pre-trained language model optimized for abstractive text summarization. |
| Approach: | They propose a pre-trained language model optimized for abstractive text summarization that uses a two-phase pre-training technique to improve model's performance. |
| Outcome: | The proposed model outperforms the competing models on low-resource summarization tasks in zero-shot and few-shot settings. |
Convolutions and Self-Attention: Re-interpreting Relative Positions in Pre-trained Language Models (2021.acl-long)
Copied to clipboard
| Challenge: | Recent work has shown that convolutions have been successful in natural language learning. |
| Approach: | They propose a convolutional approach to construct relative position embeddings in self-attention layers and propose 'compact attention' they propose multiple ways to integrate convolutions into Transformer self- attention. |
| Outcome: | The proposed composite attention improves performance on multiple downstream tasks, replacing absolute position embeddings, and is more expressive than convolutions in NLP. |
Variational Language Concepts for Interpreting Foundation Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Foundation Language Models (FLMs) have achieved remarkable success in natural language processing. |
| Approach: | They propose a variational Bayesian framework to provide word-level interpretations for FLMs . they propose valc to find optimal language concepts to interpret FLM predictions . |
| Outcome: | Empirical results show that the proposed framework can provide conceptual interpretations for foundation language models. |
Towards Efficient Visual-Language Alignment of the Q-Former for Visual Reasoning Tasks (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained large language models can be fine-tuned with instruction tuning to align the model responses with human intentions. |
| Approach: | They investigate the effectiveness of parameter efficient fine-tuning (PEFT) of the Q-Former with visual reasoning benchmarks ScienceQA and IconQA. |
| Outcome: | The proposed model achieves comparable performance to full fine-tuning using under 2% of the trainable parameters. |
XC-Cache: Cross-Attending to Cached Context for Efficient LLM Inference (2024.findings-emnlp)
Copied to clipboard
Joao Monteiro, Étienne Marcotte, Pierre-Andre Noel, Valentina Zantedeschi, David Vazquez, Nicolas Chapados, Christopher Pal, Perouz Taslakian
| Challenge: | XC-Llama uses pre-trained decoder-only models to condition generation on reference text without the prompt. |
| Approach: | They propose a model that uses cross-attention to condition generation on reference text without the prompt. |
| Outcome: | The proposed models outperform prompt-based inference methods and reduce space footprint relative to standard KV caching by two orders of magnitude. |
Training-free Deep Concept Injection Enables Language Models for Video Question Answering (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to train pretrained language models for zero-shot crossmodal tasks require crossmodal pretraining. |
| Approach: | They propose to inject visual concepts into the input text embedding space of a pretrained language model and build adaptation layers based on the intermediate representation of concepts. |
| Outcome: | The proposed model performs zero-shot crossmodal tasks without crossmodal pretraining . it is based on the injection of visual concepts as input tokens and augmentation in intermediate features . the proposed model achieves competitive or even better results in zero- shot and fine-tuning settings . |